Corrected optical property value-based search query

ABSTRACT

A corrected optical property value of an optical property of interest captured within a digital image is determined. A search query based on the corrected optical property value is determined. A database is searched using the search query to determine search results for the search query that have appearance attributes with optical properties at least similar to the corrected optical property value.

BACKGROUND

Color and other optical properties are an important consideration formany types of users. For example, a car buyer may be interested inpurchasing a car having a particular shade of red. As another example,interior decorators may want to select paint having a particulartexture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of an example method for acquiring search resultsthat match a color of interest captured within a digital image.

FIG. 2 is a flowchart of an example method for acquiring search resultsthat match a color of interest captured within a digital image that isconsistent with the method of FIG. 1 but is presented from a perspectiveof a mobile device.

FIG. 3 is a flowchart of an example method for acquiring search resultsthat match a color of interest captured within a digital image that isconsistent with the method of FIG. 1 but is presented from a perspectiveof a computing system interacting with a mobile or other device.

FIG. 4 is a diagram of an example uncalibrated digital image of a colorof interest and a physical color chart within a physical scene.

FIG. 5 is a diagram of an example architecture including the mobiledevice that performs the method of FIG. 2 and the system that performsthe method of FIG. 3.

DETAILED DESCRIPTION

As noted in the background, color and other optical properties animportant consideration for many different types of users. A car buyermay not just be interested in red cars per se, but rather a car having aparticular shade of red. Many users do not know the precise name of aparticular color of interest, and indeed, the same shade of a givencolor may be called different names by different vendors. This meansthat existing search engines, which return results responsive to textualsearch queries, can be unhelpful. For example, entering in the searchquery “red car,” will likely not reveal the exact shade of red a userhas in mind.

Techniques disclosed herein overcome these shortcomings. An uncalibrateddigital image captures a physical scene including at least an opticalproperty of interest, such as a particular color, and which can alsoinclude a physical optical property calibration chart, such as aphysical color calibration chart. A corrected optical property valuethat accurately denotes the optical property of interest within theimage is determined, such as based on the physical chart captured withinthe image as well, and an appearance-encoded search query is determinedbased on the corrected optical property value. A database can besearched using the search query to locate search results that haveappearance attributes with optical properties at least similar to thecorrected optical value.

For example, a user may want a color of paint to paint his or her housethat matches a color of a building. The user captures an image of thebuilding using his or her smartphone. Most smartphones and other typesof mobile devices capture uncalibrated digital images that do notcapture color with the required color accuracy. Even the few suchdevices that do may not be able to accurately capture the color the usersees, because of the lighting and other variables of the scene orenvironment in which a picture is being taken. Therefore, a correctedcolor value is determined that accurately denotes the color the user hasseen, such as a numeric definition of the color in a predetermined colorspace.

A search query including this corrected color value and the phrase“exterior house paint” can then be used to search a general-purposedatabase. The corrected color value may be used by itself to search aspecial-purpose database for the product of interest, such as exteriorhouse paint. The user thus receives search results of exterior housepaint colors that best match the color of the building that the user sawand captured a digital image of, from which the user may select adesired color to purchase and paint his or her house.

In some implementations, a user thus just has to bring along a smallchart when out and about to accurately identify colors or other opticalproperties of interest that the user sees. The user can use his or hersmartphone to capture an image of a discovered color or other opticalproperty of interest with the chart placed alongside. This opticalproperty of interest is convertible to a corrected optical propertyvalue that accurately describes the optical property of interest,because the calibration chart has been digitally captured under the sameconditions as the optical property of interest itself. The correctedoptical value is used as the basis upon which a search query isgenerated, which is then used to search a database to receive searchresults having appearance attributes with optical properties at leastsimilar to the corrected optical value.

The remainder of the detailed description is described in relation tocolor as an example optical property. However, the techniques disclosedherein are applicable to optical properties other than color as well.Examples of such other optical properties include texture, translucence,gloss, and pearlescence, among others.

FIG. 1 shows an example method 100. The example method 100 can beimplemented at a system. The system may be or include a computing systemthat is provided by a service provider, such as a search engine provideror another type of service provider. This computing system may interactwith devices of users, such as mobile devices like smartphones. Thesystem may include the mobile device as well. For instance, some partsof the method 100 may be performed by the computing system, and otherparts performed by a mobile or other device. Specific exampleimplementations in such respects are described later in the detaileddescription, with reference to FIGS. 2 and 3.

A corrected color value is determined from a color of interest within adigital image (102). For instance, a device such as a mobile device,like a smartphone, a standalone digital camera, and so on, can capturean uncalibrated digital image of a physical scene that includes a colorof interest and a physical color calibration chart. The mobile devicecan be a device that is not a professional color-capturing device. Thatis, the digital images that the mobile device captures are uncalibrated,in that the image-capturing mechanism of the mobile device, such as acharged coupled device (CCD) or a contact image sensor (CIS), is notable to be color calibrated or has not been color calibrated.

The physical scene is likewise uncalibrated, in that the lightingconditions and other variables that impact color rendition and accuracywithin the digital image may not be ideal, and are typically notspecified to any great degree of accuracy, if at all. For example, thephysical scene may be an outdoors scene in which a user of the mobiledevice happens to be currently located, an indoor location such as ashopping mall, and so on. The user may happen upon a physical objectthat has the color of interest, for instance, during the normal courseof his or her day.

In both these respects—from a device standpoint and from a scene orenvironmental standpoint—the digital image is said to be uncalibrated.Besides the color of interest, the digital image includes the physicalcolor calibration chart. The physical color calibration chart permitsthe color of interest to be accurately determined, insofar as the chartcan have predetermined and preselected colors and has its picture takenas part of the same image, under the same conditions and using the samedevice, as the color of interest itself.

A color chart as used herein is intended to encompass any suitablephysical color reference comprised of a set of colors selected to enablegeneration of a color profile. Such a color profile can be used, forexample, for color calibration of a device or color correction of animage. A color chart may be constituted as a flat, physical objectcolored with an arrangement of standardized color samples (e.g., aplurality of color patches). Color charts, such as the X-Rite colorchecker, may be rectangular and have a selected size and color patchlayout.

The corrected color value of the color of interest can be determinedbased on the physical color calibration chart and the color of interestas captured within the uncalibrated digital image. The corrected colorvalue is an accurate measurement of the color of interest that correctsfor the uncalibrated nature of the digital image. That is, the correctedcolor value is an accurate measurement of the color of interest eventhough the digital image was captured using an uncalibrated device andunder not precisely known lighting and other environmental conditions.For example, the corrected color value may be represented as a series ofcolor coordinates within a color scale like the CIELAB color scale, as anumber of color channel values of the constituent color channels of apredetermined color space, and so on.

Determination of the corrected color value can be achieved as describedin one or more of the previously referenced copending patentapplications. The color calibration chart is captured in the same imageas the color of interest, using the same device and under the sameenvironmental conditions. Therefore, the corrected color value of thecolor of interest can be determined from both the color of interest andthe color calibration chart as captured within the image.

A search query is determined based on the corrected color value (104).The search query can be said to be an appearance-encoded search query,in that it encodes an appearance of an object, property, or other item,since the search query is based on the corrected color value. The searchquery may be a structured query language (SQL) query, or may be a queryformatted for use with an existing general purpose or special purposesearch engine available over the Internet or otherwise.

In one implementation, the search query includes the corrected colorvalue itself. For example, if a user is searching for colors similar tothe corrected color value, the search query may include just thecorrected color value. In this respect, for instance, a SQL query can begenerated by wrapping the corrected color value within a SQL-formattedwrapper that is understandable by a SQL database. As another example, ifa user is search for a particular type of thing, such as a car, forinstance, then the search query may include both the text “car” and thecorrected color value. In this respect, for instance, a SQL query can begenerated by wrapping both the corrected color value and theaforementioned text within a SQL-formatted wrapper. The search query isappearance encoded because it includes the corrected color value.

In another implementation, the search query includes one or more naturallanguage search terms generated from the corrected color value. Thenatural language search terms may be text that represents in words theclosest color to the corrected color value that can be so represented.For example, if the corrected color value represents a shade of mauve,the search query may include the term “mauve,” and if a user issearching for cars having this color, the search query may also includethe term “car.” A SQL query may thus be generated by wrapping thenatural language search terms by themselves, or in addition to the itemhaving this appearance, within a SQL-formatted wrapper. The search queryis appearance encoded because it includes the natural language searchterms determined from the corrected color value.

Generating natural language terms from the corrected color value can bedetermined in a variety of different ways. A look-up table may beemployed that maps ranges of corrected color values to various terms.More algorithmic approaches may also be employed.

A database is searched using the search query to determine searchresults that have appearance attributes with colors matching or similarto the corrected color value encoded within the query (106). Thedatabase may be a general-purpose search engine, like those commonlyaccessible over the Internet. The database may be a special-purposesearch engine as well. For example, there may be a database of motorvehicles that denote different cars and light trucks by the precisecolors in which they are available. As another example, there may be adatabase of paint colors that denote different paints from the same ordifferent suppliers by the precise colors in which such paint isavailable.

Each search result has an appearance attribute having a color matchingor similar to the corrected color value to some degree (i.e., by morethan a threshold). For example, a user may be searching for cars havinga particular color represented by the corrected color value. Each searchresult may specify the manufacturer, year, model, and trim of a car.Each search result also has an appearance attribute that specifies thecolor of the car, which matches the corrected color value to somedegree. The more entries there are that have appearance attributesmatching or similar to the corrected color value, the closer the searchresults will be to the corrected color value encoded within the searchquery.

The search results can be ordered by the proximity of the colors oftheir appearance attributes to the corrected color value encoded withinthe search query, particularly where the search query includes thecorrected color value itself, such as CIELAB color scale coordinates orcolor channel values of color channels of a color space. For example,assume the corrected color value is abstracted as the color value X,which may itself be or include a number of color channel values. Thesearch results are thus ordered by how closely their color values are toX in this example.

The search results can be ordered by how closely they are tagged withnatural language terms matching the natural language search terms of thesearch query generated from the corrected color value. For example,assume the search query contains the natural language search terms “deepmauve.” A search result tagged with the natural language terms “darkmauve” is likely to then be ordered as more close to the search querythan a search result tagged with the natural language terms “lightmauve.”

Different types of techniques are thus amenable to construction of thesearch query and the ordering of the search results. A complex querythat integrates different optical property values may yield searchresults that are sorted using a technique for integrating different suchmetrics, such as Kalman filtering. Dynamic user weighting of differentoptical property values—i.e., different appearance attributes—may beemployed to ensure that the search results accurately reflect theappearance encoding within the search query where the search queryincludes multiple optical property values.

The search results may be filtered or sorted in other ways as well, suchas based on retail and commerce constraints, item location, quantitiesavailable, special offers, and other factors. For example, a user may beinterested in purchasing a new car having a particular color. In thisrespect, search results that are for cars have the right color but thatare not currently manufactured, or are not in stock at dealerships nearthe user, may be excluded.

The search results are output (108). For instance, a user initiating asearch on his or her mobile device may view the search results on thedevice. The search results may be output in other ways as well. As oneexample, a user interesting in purchase a car of a desired color ofinterest may have the search results sent to selected dealerships thathave such vehicles on the lot, which may then contact the user to set upa test drive, send the user a physical letter or an electronic mail withthe requested information on the product with the requested color, andso on.

FIG. 2 shows an example method 200 that is consistent with the method100 but that is from the perspective of a mobile device of a user, likea smartphone. The method 200 may be implemented as computer-readablecode executable by a processor of the mobile device. Thecomputer-readable code can be stored on a non-transitorycomputer-readable data storage medium of the mobile device.

The mobile device captures a digital image of a physical scene includinga color of interest and a physical color calibration chart (202). Themobile device determines a corrected color value of the color ofinterest (204). In one implementation, the mobile device computes thecorrected color value itself. In this implementation, the mobile devicethus includes the logic for computing the corrected color value from thecolor of interest and the chart as captured within the image.

In another example implementation, the mobile device is communicativelyconnected to a computing device like a server computing device, and moregenerally a system, over one or more networks such as mobile networkslike 3G and 4G mobile networks, the Internet, and other types ofnetworks. In this implementation, the mobile device can transmit theuncalibrated digital image to the computing device over the network. Thecomputing device includes the logic for computing the corrected colorvalue from the color of interest and the chart as captured within theimage, and thus computes the corrected color value from the imagereceived from the mobile device. The computing device may return thecorrected color value back to the mobile device, which thus receives thecorrected value from the system.

The mobile device is said to determine search results obtained by adatabase being searched using a search query based on the correctedcolor value (206). In one implementation, the mobile device may itselfgenerate the search query as has been described. In this implementation,the database may be preloaded into the mobile device, in which case themobile device can perform the search itself to generate the searchresults. As another example, the mobile device may query a database thatis located external to the device, such as by transmitting the searchquery to a search engine that acts as a front end for the database, andreceiving back in response the search results.

In another implementation, the mobile device may determine the searchquery using the aforementioned system. For example, the mobile devicemay transmit the digital image to the system, which directly determinesthe corrected color value and the search query. As another example, themobile device may transmit the corrected color value to the system,where the corrected color value has been determined either directly bythe mobile device or by the same or different system. In either example,the mobile device may receive the search query back from the system suchthat the mobile device effectuates the search of the database, or thesystem itself may conduct the search of the database and return thesearch results back to the mobile device.

Therefore, in varying implementations the mobile device can perform upto all aspects of the method 100 that has been described. For instance,if the mobile device includes the logic to perform each part of themethod 100, then the method 200 is executed by the mobile device withoutassistance from any external or remote system. At the other extreme, themobile device may just capture the digital image and display the searchresults to the user, such that it is said the device determines thecorrected color value, the search query, and/or the search results withthe assistance of an external system. For example, the mobile device insuch an implementation may capture the digital image, determine thecorrected color value, and/or determine the search query, but transmitthe search query to an external system that acts as the front end of thedatabase for the actual performance of the search.

FIG. 3 shows an example method 300 that is consistent with the method100 but that is from the perspective of a computing system. Thecomputing system determines a search query based on a corrected colorvalue determined from a color value of interest and a color calibrationchart captured within a digital image (302). The digital image can bereceived from a device like a mobile device. The system may receive justthe digital image from the device, and determine the corrected colorvalue itself, or the system may receive just the corrected color valuefrom the device without receiving the digital image.

The computing system searches a database using the search query todetermine search results (306), as has been described. The database maybe stored on a storage device of the computing system, such as astorage-area network (SAN) or other type of storage device, and thesystem may search the database by querying the database itself. Inanother implementation, the system may transmit the search query to adifferent system, such as a system running a general-purpose searchengine, and receive the search results back from this other system. Ineither case, however, it is said that the system itself searches thedatabase, either directly or indirect.

The computing system returns the search results to the device from whichthe digital image and/or the corrected color value has been received(306). The system in this respect may be operated by a service provideras a service. The service may be provided for free to device users,where, for instance, advertisers and/or parties selling or manufacturingitems within the database pay the provider. The service may be providedby charging the users of the devices in another implementation as well.

FIG. 4 shows an example uncalibrated digital image 400 captured by amobile device like a smartphone. A user may in the course of his or herdaily life encounter a physical object like a vase 402 sitting on atable 404, and determine that the vase 402 has a color that the userthinks would be perfect for a particular project he or she is workingon. That is, the color of the vase 402 is the color of interest.Therefore, the user may remove the physical color calibration chart 406that he or she keeps in a wallet or a purse, and places it on the table404 against the vase 402. The user may also hold up the chart 406 by oragainst the vase 402. The user then uses his or her mobile device tosnap a picture of the resulting physical scene to capture the digitalimage 400.

FIG. 5 shows an example system 500. The system 500 can include one ormore of a mobile device 502, a computing system 504, and a search engine506, which are communicatively connected to one another via a network507. The network 507 may be or include mobile networks, the Internet,intranets, extranets, and other types of networks as well.

The mobile device 502 includes at least a processor 508, networkhardware 510, and a non-transitory computer-readable data storage medium512. The mobile device 502 can include an image-capturing mechanism,such as a CCD or a CIS, as well. The network hardware 510 can include awireless transceiver to permit the mobile device 502 to communicate overthe network 507. The computer-readable data storage medium 512, whichmay be nonvolatile semiconductor memory, stores computer-readable code514 that is executable by the processor 508. The code 514 implementslogic by which the mobile device 502 performs the method 200 that hasbeen described.

The computing system 504 can be realized as one or more computingdevices, such as servers. As such, the computing system 504 includes aprocessor 516, network hardware 518, and a non-transitorycomputer-readable data storage medium 520. The network hardware 518 maybe or include a network adapter by which the system 504 communicatesover the network 507. The computer-readable data storage medium 520,which may include semiconductor memory and/or magnetic media, storescomputer-readable code 522 executable by the processor 516. The code 522implements the logic by which the system 504 performs the method 300that has been described.

In some implementations, the computing system 504 can include a storagedevice 524, which may be the same or a different storage than thenon-transitory computer-readable data storage medium 520. The storagedevice 524 stores a database 526. In such implementations, the database526 is that which the system 504 searches the search query for toacquire search results for the search query.

In implementations in which the system 500 includes the search engine506, the search engine 506 also includes or has access to a database528, against which the mobile device 502 and/or the system 504 submitsthe search query to acquire search results. The search engine 506 may beimplemented over one or more computing devices, such as servers. Thesearch engine 506 may be operated by a different provider than theservice provider which operates the computing system 504. In someimplementations, the system 500 includes both the database 526 at thesystem 504 and the database 528 at the search engine 506, whereas inother implementations, the system 500 includes just the database 526 orjust the database 528.

The system 500 thus illustrates that there can be up to three differenttypes of parties via which techniques disclosed herein are operable. Endusers operate mobile devices like the mobile device 502. One type ofservice provider operates the computing system 504. A different type ofprovider can operate the search engine 506. For example, the searchengine 506 may be a general-purpose search engine accessible over theInternet.

We claim:
 1. A system comprising: networking hardware to receive from a device to which the networking hardware is communicatively connected one of: a digital image of a physical scene including an optical property of interest and a physical optical property calibration chart; a corrected optical property value of the optical property of interest determined based on the physical optical property calibration chart and the optical property of interest within the digital image; a processor; and a non-transitory computer-readable data storage medium storing computer-readable code executable by the processor to: if the networking hardware receives the digital image and not the corrected optical property value, determine the corrected optical property value based on the physical optical property calibration chart and the optical property of interest within the digital image; determine a search query based on the corrected optical property value; search a database using the search query to determine search results for the search query that have appearance attributes with optical properties at least similar to the corrected optical value; and return the search results to the device via the networking hardware.
 2. The system of claim 1, wherein the networking hardware is to receive from the device just the digital image and not the corrected optical property value, and the computer-readable code is executable by the processor to further determine the corrected optical property value of the optical property of interest based on the physical optical property calibration chart and the optical property of interest within the digital image.
 3. The system of claim 1, wherein the networking hardware is to receive from the device just the corrected optical value as determined by the device and not the digital image.
 4. The system of claim 1, further comprising: a storage device storing the database, wherein the computer-readable code is executable by the processor to search the database by directly querying the database using the search query.
 5. The system of claim 1, wherein the database is stored on a different system managed by a search provider different from a service provider managing the system, wherein the computer-readable code is executable by the processor to search the database by transmitting the search query to the different system via the networking hardware and by receiving from the different system in response the search results via the networking hardware.
 6. The system of claim 1, wherein the computer-readable code is executable by the processor to determine the search query as including the corrected optical property value.
 7. The system of claim 6, wherein the computer-readable code is executable by the processor to search the database to determine the search results as ordered according to proximity of the optical properties of the appearance attributes of the search results to the corrected optical property value.
 8. The system of claim 1, wherein the computer-readable code is executable by the processor to determine the search query as including one or more natural language search terms from the corrected optical property value.
 9. The system of claim 8, wherein the computer-readable code is executable by the processor to search the database to determine the search results ordered according to closeness matching of natural language terms within the search results to the one or more natural language search terms.
 10. A non-transitory computer-readable data storage medium executable by a mobile device having image-capturing capability to perform a method comprising: capturing a digital image of a physical scene including an optical property of interest and a physical optical property calibration chart; determining a corrected optical property value of the optical property of interest based on the physical optical property calibration chart and the optical property of interest as captured within the digital image; determining search results obtained by a database being searched using a search query based on the corrected optical property value, the search results having appearance attributes with optical properties at least similar to the corrected optical property value; and outputting the search results.
 11. The non-transitory computer-readable data storage medium of claim 10, wherein determining the corrected optical property value comprises: transmitting the digital image to a system to which the mobile device is communicatively connected; and receiving the corrected optical property value, as determined by the system, from the system.
 12. The non-transitory computer-readable data storage medium of claim 10, wherein determining the corrected optical property value comprises computing the corrected optical property value at the mobile device itself.
 13. The non-transitory computer-readable data storage medium of claim 10, wherein determining the search results comprises generating the search query at the mobile device itself.
 14. The non-transitory computer-readable data storage medium of claim 10, wherein determining the search results comprises receiving the search query from a system to which the mobile device is communicatively connected after the mobile device has transmitted the digital image and/or the corrected optical property value to the system.
 15. The non-transitory computer-readable data storage medium of claim 10, wherein determining the search results comprises searching the database as preloaded into the mobile device, by the mobile device.
 16. The non-transitory computer-readable data storage medium of claim 10, wherein determining the search results comprises receiving the search results from a system to which the mobile device is communicatively connected after the mobile device has transmitted one or more of the digital image, the corrected optical property value, and the search query to the system. 